Among the GHA tests, it looks like the windows job gets the farthest, reaching (but not completing) test_init_with_file.jl. It could be that that is an artifact download issue (maybe even on windows?). Hm, but macOS got that far on the one before that 🤔. Ah, we have timeout-minutes: 30 for CI, I'm fine bumping that to 1 hr. It's probably just taking longer due to slower compilation time.

On buildkite, it looks like some jobs are getting precompiled twice, ah this is likely due to using a fixed old version of atmos. Developing atmos in the perf env should fix this, I think.

We can increase the ref counter to fix the reproducibility tests, could someone review the physics changes?

Finally, the MPI job looks like it may be running out of memory, could we increase the mem request for test restart MPI?

I am finding that ClimaCore 0.14.12 is fine with Julia 1.10, but we need 0.14.19 for Julia 1.11 (https://github.com/CliMA/ClimaAtmos.jl/actions/runs/12694181365/job/35383420600).

Unfortunately, I don't see any easy way to specify this constraint, so maybe the safest is just bumping the minimum to 0.14.19

I am finding that ClimaCore 0.14.12 is fine with Julia 1.10, but we need 0.14.19 for Julia 1.11 (https://github.com/CliMA/ClimaAtmos.jl/actions/runs/12694181365/job/35383420600).

Unfortunately, I don't see any easy way to specify this constraint, so maybe the safest is just bumping the minimum to 0.14.19

That makes sense. We kind of got bit by JuliaLang/julia#56295. The main branch is fine, but the high-level dispatch that effectively fixed this issue must have been merged in 0.14.19.

I don't know what we could have done to prevent this, unfortunately.